Image blending based on image reference information

ABSTRACT

A target image is obtained from a cloud database based on reference information associated with a source image that is captured by an image capture device. The target image is blended with the source image to generate a blended image.

BACKGROUND

Photographs and other images captured by digital cameras can have highlevels of image noise. As used herein, image noise refers to variations(usually unwanted) in brightness, color and/or other aspects of imagequality in images produced by a digital camera or other image capturedevice. High levels of image noise can affect an image to the pointwhere a user decides not to save, share or print the image.

BRIEF DESCRIPTION OF DRAWINGS

The following description includes discussion of figures havingillustrations given by way of example of implementations of embodimentsof the invention. The drawings should be understood by way of example,not by way of limitation. As used herein, references to one or more“embodiments” are to be understood as describing a particular feature,structure, or characteristic included in at least one implementation ofthe invention. Thus, phrases such as “in one embodiment” or “in analternate embodiment” appearing herein describe various embodiments andimplementations of the invention, and do not necessarily all refer tothe same embodiment. However, they are also not necessarily mutuallyexclusive.

FIG. 1 is a block diagram illustrating a system according to variousembodiments.

FIG. 2 is a block diagram illustrating a system according to variousembodiments.

FIG. 3 is a flow diagram of operation in a system according to variousembodiments.

DETAILED DESCRIPTION

Various embodiments described herein facilitate image registration andblending using high quality images maintained in a cloud database orother network database.

Image registration is the process of transforming different sets of data(e.g., image data) into a common coordinate system. Data used for imageregistration may be derived from multiple different digital photographs,different sensors, etc. By bringing image data together into a commoncoordinate system, image registration facilitates image blending.

For example, an image capture device, such as a digital camera, may beused to capture multiple images of a scene. If the quality of thecapture device and/or the image is low, image registration can beperformed on the multiple images. After image registration, several orall of the multiple images can be blended to reduce image noise in thefinal blended image. However, if all of the images used for blendingwere captured by the same low-quality capture device, the improvement inthe final blended image may be minimal.

Accordingly, embodiments described herein use database of imagescaptured by various devices (e.g., digital cameras, camera phones,smartphones, video cameras, etc.) to improve the overall quality of animage captured by a particular image capture device. In particular, theimage capture device tags or associates images with referenceinformation (e.g., location, device orientation, magnetic direction,etc.) that can be used to find one or more matching images in thedatabase. Image registration is performed on at least one matching imagefrom the database and the database image is subsequently blended withthe original image to improve one or more image qualities of theoriginal image.

FIG. 1 is a block diagram illustrating a system according to variousembodiments. FIG. 1 includes particular components, modules, etc.according to various embodiments. However, in different embodiments,other components, modules, arrangements of components/modules, etc. maybe used according to the teachings described herein. In addition,various components, modules, etc. described herein may be implemented asone or more software modules, hardware modules, special-purpose hardware(e.g., application specific hardware, application specific integratedcircuits (ASICs), embedded controllers, hardwired circuitry, etc.), orsome combination of these.

In FIG. 1, image capture device 100 may be a digital camera, a cellphone camera, smartphone camera, a video camera or any other devicecapable of converting optical images to digital images. Optical system102 produces optical images that are captured and converted to digitalimages by image sensor 110. Reference sensor module 120 acquiresexternal sensory data in connection with image capture by image sensor110. Reference sensor module 120 may be an orientation sensor (e.g.,accelerometer, gyroscope, etc.), a magnetometer, a location sensor(e.g., GPS), or other sensor capable of providing information thatdescribes the position, location and/or orientation of image capturedevice 100 with respect to one or more reference axes. In certainembodiments, reference sensor module 120 may be a combination of variousreference sensors such as, but not limited to, those described above.For example, in some embodiments, reference sensor module 120 maycollect sufficient information to describe the orientation of imagecapture device 100 on a six axis basis (e.g., horizontal position,vertical position, rotation on the z-axis, yaw, pitch and roll). Toaccomplish this, reference sensor 120 may additionally include GPSlocation information, time and date of image capture, as well as variousexposure-related details (e.g., white-balance levels, focus position,etc.) for a captured image.

Communications module 130 sends an image captured by device 100(referred to herein as either a source image or a reference image) alongwith associated external sensory information (e.g., the referenceinformation described above) to a processing entity via a networkconnection. Communications module 130 can be any type of module capableof communication with an external network. For example, communicationsmodule 130 could be a LAN (local area network) module, such as anEthernet module. In various embodiments, communications module is awireless communications module. Examples of wireless communicationsmodules include WLAN (wireless local area network) modules, cellularradio modules (e.g., 2G, 3G, WiMax, Ultra Mobile Broadband, 3GPP LongTerm Evolution, 4G, etc.) and the like.

In response to sending a source image with associated external sensorydata to a processing entity, communications module 130 subsequentlyreceives a blended version of the image from the processing entity andstores it in memory 140. The blended version of the image is generatedby blending the original source image with a similar image selected froma database based on the external sensory data. For example, a user ofimage capture device 100 may capture an image of the Eiffel Tower inParis, France. In conjunction with the image capture, the Eiffel Towerimage is tagged with reference information (e.g., GPS location, time ofcapture, magnetic direction of capture, orientation, etc.) by referencesensor module 120. Based on the reference information, a similar imageof the Eiffel Tower is selected from the database (e.g., captured from asimilar location, at a similar time, with a similar camera orientation,etc.). The database image of the Eiffel Tower may then be transformed(e.g., via image registration) such that at least one element of thedatabase image aligns with a common element of the original image. Forexample, image registration may result in the tower from the databaseimage aligning with the tower in the original image. Once one or moreimage elements are aligned, the database image and the original imageare blended to generate a blended version of the original that hasimproved image quality characteristics (e.g., white-balance, color,focus, etc.) and/or reduced image noise.

FIG. 2 is a block diagram of a system according to various embodiments.FIG. 2 includes particular components, modules, etc. according tovarious embodiments. However, in different embodiments, othercomponents, modules, arrangements of components/modules, etc. may beused according to the teachings described herein. In addition, variouscomponents, modules, etc. described herein may be implemented as one ormore software modules, hardware modules, special-purpose hardware (e.g.,application specific hardware, application specific integrated circuits(ASICs), embedded controllers, hardwired circuitry, etc.), or somecombination of these.

In FIG. 2, image capture device 200 may be a digital camera, a cellphone camera, smartphone camera, a video camera, or any other devicecapable of converting optical images to digital images. Optical system202 produces optical images that are captured and converted to digitalimages by image sensor 210. Reference sensor module 220 acquiresexternal sensory data in connection with image capture by image sensor210. Reference sensor module 220 may be a single sensor (e.g., anorientation sensor, a magnetometer, a location sensor, etc.), or it maybe comprised of multiple sensors capable of providing information thatdescribes the position, location and/or orientation of image capturedevice 200. For example, in some embodiments, reference sensor module220 may collect information to describe the orientation of image capturedevice 200 on a six axis basis (e.g., horizontal position, verticalposition, rotation on the z-axis, yaw, pitch and roll). In otherembodiments, reference sensor module 220 collects information todescribe orientation using fewer than six axes.

In addition to the orientation information described above, in variousembodiments, reference sensor module 220 includes GPS locationinformation, time and date of image capture, and/or variousexposure-related details (e.g., white-balance levels, focus position,etc.) for a captured image.

Communications module 230 can be any module capable of communicatinggenerally with cloud 280, or more specifically, with processing entity250, and/or database 270 (either directly or indirectly). Cloud 280 isrepresentative of various forms of Internet-based computing. In certainembodiments, cloud 280 could be representative of a private networkconnection to processing entity 250 and/or database 270.

In an example, communications module 230 sends an image captured bydevice 200 (referred to herein as either a reference image or a sourceimage) along with reference information for the source image acquired byreference sensor module 230 to cloud 280 where it is subsequentlyreceived by receiver 252 on processing entity 250. Processing entity 250can be a server, a plurality of servers, or any other cloud-connecteddevice capable of processing image information.

Query module 262 queries database 270 using the received referenceinformation for the source image to find a matching image. In variousembodiments, database 270 contains high quality images that, whenblended with a source image, are likely to reduce the image noise in thesource image. Of course, it is not necessary that embodiments describedherein maintain a quality threshold for images stored in database 270.

In an example, processing entity 250 receives a source image from imagecapture device 200 taken of the Eiffel tower in Paris, France. Thereference information included with the source image contains the GPSlocation, time and date associated with the image capture. In addition,the reference information includes the magnetic direction andorientation of image capture device 200 at the time the source image wascaptured. Query module 262 sends a query to database 270 based on thereceived reference information. While the exact criteria for selecting amatching target image may vary according to circumstances, database 270returns a target image that matches (e.g., within a threshold) thereference information in the query. In other words, the retrieved targetimage is also an image of the Eiffel tower, taken from the same location(e.g., within a threshold) and facing the same direction (e.g., within athreshold).

Based on the time and date information of the source image, the targetimage may have similar lighting characteristics. For example, if theoriginal source image was taken at sunset in July, database 270 mightreturn a target image taken at nearly the same time during the samemonth (though perhaps not in the same year). Database 270 may also use atable or computer-coded process to determine the time of sunset indifferent months to find other target images taken at sunset to increasethe likelihood of finding a target image whose reference informationmatches that of the source image.

Though the example described above uses GPS location, magneticdirection, time and date to find a suitable target image, it should beunderstand that more, less, or different reference information, such asdescribed herein, can be used in matching the source image with a targetimage. In various embodiments, the accuracy of matching results mayincrease as the quantity and/or quality of reference informationincreases.

Image registration module 254 performs image registration on the targetimage returned from database 270. In particular, the target image istransformed to align with the source image. For example, a query ofdatabase 270 may return a target image captured from a slightlydifferent location than the source image. However, by transforming theimage (e.g., shifting, scaling, etc.) to align with the source image(e.g., aligning the tower in a target image of the Eiffel tower and in asource image of the Eiffel tower), the two images can be combined and/orblended to generate an enhanced final image. Image registration may beintensity-based or feature-based. Intensity-based methods compareintensity patterns in images via correlation metrics, whilefeature-based methods find correspondence between image features such aspoints, lines, and contours.

Blending module 256 uses the transformed target image from imageregistration module 254 to blend with the original source image capturedby image capture device 200. Various types of blending and/or blendingmodes may be used by blending module 256. For example, blending module256 may perform dodging, burning, division, addition, subtraction,darkening, lightening, luminosity adjustments, and/or hue, saturation,color adjustments, etc. In some embodiments, more than one target imagemight be retrieved from database 270 and used for blending with theoriginal source image.

The blending results from blending module 256 are sent via transmitter252 back to image capture device 200 where they are received bycommunications module 230. In various embodiments, the user of imagecapture device 200 is shown a preview of the blended image on a display(e.g., LCD display) of the device and is given the opportunity to savethe blended image in memory 240. Accordingly, in embodiments where imagecapture device 200 is a low quality capture device, a user is able toenhance the quality of various images captured by device 200 using thefeatures described herein.

Various modules and/or components illustrated and described in FIG. 2may be implemented as a computer-readable storage medium containinginstructions executed by a processor (e.g., processor 258) and stored ina memory (e.g., storage 260). Additionally, the various modules ofprocessing entity 250 could be implemented directly on image capturedevice 200 as processing entity 242.

FIG. 3 is a flow diagram of operation in a system according to variousembodiments. FIG. 3 includes particular operations and execution orderaccording to certain embodiments. However, in different embodiments,other operations, omitting one or more of the depicted operations,and/or proceeding in other orders of execution may also be usedaccording to teachings described herein.

A processing entity receives 310 a source image captured by a device(e.g., a digital camera, cell phone camera, smartphone camera, videocamera, etc.). The source image may be a still image or it may be aframe or group of frames belonging to a video segment. The source imageis received via a network connection (e.g., Ethernet, LAN, WLAN, etc.).The reference information may include location, orientation, direction,time/date of capture, and/or other information used to describe theposition and/or conditions of the camera when the source image wascaptured.

Based on the reference information, a target image is obtained 320 froma database. For example, the target image may be selected because it wassimilarly captured from the same location, with the camera pointed inthe same direction, and with the camera having the same orientation(e.g., on one or multiple axes). In certain embodiments, some types ofreference information may be given more weight in selecting a targetimage. For example, location information might be given more weight thandate information.

The selected target image is blended 330 with the source image togenerate a blended image. In various embodiments, the blending comprisesimage registration followed by image blending. In other words, at leastone element of the target image is transformed to align with a commonelement in the source image. For example, the target image may betransformed such that a particular line in the target image is broughtinto alignment with a similar line in the source image (e.g., theoutline of the tower in an image of the Eiffel tower). By transformingthe target image to align with the source image, the two images may beblended according to various blending techniques and/or modes, such asthose described herein.

In certain embodiments, the processing entity responsible for blendingmay determine a subject in the source image that is not contained in thetarget image. Using the Eiffel tower example described previously, thesource image may include the photographer's friend standing prominentlyin front of the Eiffel tower. If the friend is not included in thetarget image, the processing entity may preclude the image elementsrepresenting the friend in the source image from being blended with thetarget image. In this way, the friend in the source may be retained inthe final blended image. Of course, in other embodiments, a subjectfound in the source image that is not contained in the target imagecould be included in the blending (e.g., to reduce image noise) whilestill being preserved as a subject in the final blended image.

In a situation where the target image contains a subject that is notincluded in the original source image, the elements of the subject inthe target image may be deleted from the final blended image. Forexample, if the target image contains a person in the foreground of anEiffel tower image, that person could be deleted from the final blendedimage given that the person wasn't part of the photographic scene in thesource image.

The final blended image is provided 340 via network connection to thedevice that captured the original source image. In some embodiments, thefinal blended image could be sent to a device or entity other than theoriginal capturing device.

Various modifications may be made to the disclosed embodiments andimplementations of the invention without departing from their scope.Therefore, the illustrations and examples herein should be construed inan illustrative, and not a restrictive sense.

1. A method, comprising: obtaining a target image from a cloud databasebased on reference information associated with a source image capturedby an image capture device; and blending the target image with thesource image to generate a blended image.
 2. The method of claim 1,wherein the reference information comprises external sensory informationassociated with capturing the source image.
 3. The method of claim 2,wherein the external sensory information includes information selectedfrom the group consisting of: horizontal position of the device,vertical position of the device, magnetic direction of the device,rotation on the z-axis of the device, device yaw, device pitch, deviceroll.
 4. The method of claim 1, wherein the blended image is providedvia a wireless network connection.
 5. The method of claim 1, whereinblending the target image with the source image comprises: transformingat least one element of the target image to align with a common elementof the source image to produce a transformed version of the targetimage; and blending the transformed version of the target image with thesource image to generate the blended image.
 6. An apparatus, comprising:an image sensor to capture a reference image; at least one referencesensor to acquire external sensory data associated with capturing thereference image; a communications module to send the reference image andthe external sensory data associated with capturing the reference imageto a processing entity via a network connection; the communicationsmodule further to receive a blended version of the reference image fromthe processing entity via the network connection, wherein the blendedversion of the reference image is generated by blending the referenceimage with a similar image selected from a database based on theexternal sensory data, and wherein the similar image is transformedprior to blending such that at least one element of the similar image isaligned with a common element in the reference image; and a memory tostore the blended version of the reference image.
 7. The apparatus ofclaim 6, wherein the at least one sensor is: a magnetometer; a locationsensor; or an orientation sensor.
 8. The apparatus of claim 6, whereinthe communications module is a wireless communication module.
 9. Acomputer-readable storage medium containing instructions that, whenexecuted, cause a computer to: receive a source image captured by adevice and reference data for the source image via a network connection;select a target image from a database based at least in part on thereference data; blend the target image with the source image to generatea blended image; and send the blended image to the device via thenetwork connection.
 10. The computer-readable storage medium of claim 9,wherein the reference data includes at least one of a magnetic directionof the device, an orientation of the device, a focus position of thedevice, or a geographic location of the device associated with capturingthe source image.
 11. The computer-readable storage medium of claim 9,wherein the instructions that cause the blending comprise furtherinstructions that cause the computer to: alter at least one element inthe target image to align with a common element in the source image; andblend elements of the altered target image with elements of the sourceimage to generate the blended image.
 12. The computer-readable storagemedium of claim 9, comprising further instructions that cause thecomputer to: determine a subject contained in the source image that isnot contained in the target image; preclude elements of the subject frombeing blended with the target image; and include the subject in theblended image.
 13. The computer-readable storage medium of claim 9,comprising further instructions that cause the computer to: determine asubject contained in the target image that is not contained in thesource image; delete elements of the subject from the blended image.